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• Rocket propulsion test facility 

- T est rocket engines and engine components 

• Current test program 

- El : Integrated Powerhead Demonstrator (IPD) 

- E2: Shuttle ice frost panel testing 

- E3 : HMTP (hybrid testing) 
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• What 

- Fire 

• Where 

- North side of the E-l Test stand 

• When 

- Saturday February 21, 2004, at approximately 10:08 PM 

- During structural welding operations to accommodate a new 
propellant tank 

• Impacts 

- Four-man crew vacated the area 

- Fire Department called to extinguish the fire 

- No injuries to personnel 

- Minimal damage to test stand electrical wiring 

- Schedule impact to repair 
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The Mishap Investigation Team (MIT) sought to identify 
possible proximate causes: 

• A damaged welding lead 

• Hot welding slag igniting a plastic tarpaulin folded and 
laying near a cable tray 

• Hot slag coming into direct contact with the wiring inside 
the cable tray 

• Sparks igniting a canvas tarp hanging on the outside of 
the test stand used for wind break 

* On the night of the fire it was pulled together near the cable tray 
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El Test Stand Mishap 
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• Investigation results (Fire Origin Investigation Team) 

• Hot welding slag igniting a plastic tarpaulin determined to 
be most likely cause 

• A damaged welding lead 

• Ruled out by welding lead inspections 

• No damage was found 

• Hot slag coming into direct contact with the wiring inside 
the cable tray 

• MIT could not rule this out 

• Sparks igniting a canvas tarp 

• MIT could not rule this out 
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El Test Stand Mishap 


El Test Stand (Looking South) 
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El Test Stand Mishap 


Green Tarp used as wind break 
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El Test Stand Mishap 
F indings/Recommendations 

Finding # 1 The MIT found that the hot work permit process was not followed. After 
further investigation it was determined that the utilization of an incomplete hot work 
permit had become standard practice in the test complex. 

Rationale: The hot work permit is issued by the Fire Department and the process 
requires that the work site be inspected to ensure a safe hot work environment, in this 
case to perform welding. The Fire Department is supposed to identify the precautions 
that are to be taken before work begins. In recent years, apparently to meet schedule 
demands, the Fire Department has sometimes issued hot work permits without 
identifying precautions and instead of issuing the hot work permit for a specific 
location has issued the permit for a general area. 

Recommendation # 1: 

Ensure that all personnel associated with hot work are aware of the hot work permit 
process and associated requirements. Clearly identify responsibilities including who 
verifies that precautions have been taken. Ensure a system is in place that will 
continually verify requirements are being met. 
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El Test Stand Mishap 
F indings/Recommendations 

Finding # 2: The welders were not adequately trained in the procedures for conducting 
“hot” work and they were not aware of MSS’ s written procedures for welding. 

Rationale: During the MIT’s site inspection and the subsequent interviews, the MIT 
found the following shortcomings: 

* There was no “fire watch” conducted as required by the MSS welding 
procedure. 

* Adequate fire suppression equipment was not available on the test stand as 
required by the MSS welding procedures. 

* There was no evidence of a site inspection and consequently flammables (tarps, 
cable tray and wiring, wood, and rags) were not properly protected or removed 
prior to welding operations. 

* Not enough fire protection blankets to protect the area. 

Recommendations: Ensure all personnel are trained to the proper procedure and that 
they are following work procedures. 
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F indings/Recommendations 
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Finding # 3: The fire hydrants, emergency eye wash system, and the fire suppression 
system in the signal conditioning building associated with the E-l Test Stand did not 
have water. 

Rationale: A portion of the water system in the E-complex was configured for freeze 
protection on Jan. 15, 2004. Neither the Fire Department personnel interviewed nor 
the Chief were aware that the fire fighting water was not available at E-l although an 
E-l work document indicated that the Fire Department had been notified on Jan. 15, 
2004. 

Recommendation #3 a: Implement and enforce a policy prohibiting the indefinite 
suspension of fire suppression systems without senior NASA management approval. 
Also, when water is not available, implement a system that ensures safe ‘hot’ work 
operation. 

Recommendation #3b: Improve the notification process to assure that the Fire 
Department is aware anytime that a fire suppression system is taken out-of-service and 
that the Fire Department is provided a date when service will be restored. 


17 


04/20/05 



jo 

m 


m 

> 

(/> 

m 

o 


0 ) 

Q. 

Q. 

O 

O 

c 

3 

CD 

3 


3 

a) 

< 

C7 

CD 

O 

c 

w 

o 

CD 

CD 

< 

SL 

E 

0 ) 
f I 

CD 

T5 


o 

c 

w 

CD 



El Test Stand Mishap 
F indings/Recommendations 
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Finding # 4: MSS management failed to train supervisory personnel in their 
roles and responsibilities. MSS management overloaded supervisory personnel 
with non- supervisory responsibilities. 

Rationale: The following management/supervision deficiencies were found: 

* MSS supervision is not aware of their responsibilities, is not familiar 
with work procedures, and lacks the time to visit the work site. 

* The onsite MSS safety representative failed to adequately check the 
conditions at the work site. 

* The MIT found that craft supervision personnel are being given non- 
supervisory project responsibilities which impact their ability to perform 
supervisory tasks. The role of the supervisor is critical in the 
implementation of the safe performance of the job. The inability of the 
supervisor to perform his supervisory role was found to be one of the root 
causes of the incident. 
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F indings/Recommendations 
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Recommendation # 4a: Ensure all MSS personnel including supervisory 
personnel understand their authority, responsibilities, and limits and that they 
are trained and have adequate time to perform their jobs. 

Recommendation # 4b: Analyze workloads of MSS supervisory personnel 
(and others as appropriate) to ensure adequate time is available to perform 
safety aspects of their jobs including time to become thoroughly 
knowledgeable of job requirements and procedures, frequently visit works 
sites, supervise work crews, review hazardous operations with employees, 
ensure personnel are properly trained, and be aware of physical and mental 
health of employees which could affect safe job performance. 
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F indings/Recommendations 
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Finding #5: Fire pull boxes located on the upper levels of the test stand when pulled 
failed to alert the Fire Department of the fire. 

Rationale: The fire alarm pull boxes located on the upper levels of the test stand were 
installed to report hydrogen fires and were consequently wired to the Test Control 
Room. There appearance, however, caused them to be mistaken for regular fire alarm 
pull boxes. 

Recommendation # 5a: An obvious distinction needs to be made between facility fire 
alarm system pull boxes and all other pull boxes. Also, evaluate the adequacy of the 
existing fire alarm system. 

Recommendation # 5b: Evaluate the design/design review process to ensure the 
adequacy of the process, that sufficient time is allocated for review, and that the correct 
disciplines are involved in the review including human factors. 
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F indings/Recommendations 
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Finding # 6: Inappropriate tarps were used for a windbreak at the site of the E-l 
Test Stand welding operations. 

Rationale: The tarp material was analyzed at S SC’s Material Analysis Laboratory 
and offsite at Southwest Research Institute and the material was determined to 
contain significant amounts of diesel fuel. The tarp material is considered to be 
highly flammable. 

Recommendation # 6: Specifically with respect to the tarps, immediately stop using 
the tarps in enclosed areas and in locations where hot work operations are taking 
place. 


21 


04 / 20/05 



jo 

m 

r - 

m 

> 

(/> 

m 

o 


0 ) 

Q. 

Q. 

O 

n 

c 

3 

CD 

3 


3 

a) 

< 

C7 

CD 

O 

c 

w 

o 

CD 

CD 

< 

SL 

E 

0 ) 
f I 

CD 

"O 


o 

c 

w 

CD 


El Test Stand Mishap 
F indings/Recommendations 


Finding # 7: NASA personnel accepted the use of an inadequate hot flame permit 
process. 

Rationale: The NASA’s PTD management of the work in the “E” Test Complex has 
for many years emphasized schedule. The workforce looked for shortcuts when 
being constantly reminded that meeting schedule was the most important factor. This 
was not being done to purposefully cause safety to be compromised, but the effect 
was to cause the workforce to put schedule ahead of other considerations. 

Recommendation: # 7: Ensure the implementation of all safety related procedures. 
Any deviations must get adequate review and approval. Further more, all emphasis 
on schedule must be accompanied with an even greater emphasis on safety. This 
emphasis on safety must be real. If the contractor is penalized for missing schedule 
because of safety concerns NASA must be aware of the message being sent. 
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